Generating reference files
In the Advanced tab of the Google Cloud Storage target endpoint, you can enable the Generate a reference file option. The Reference File contains a list of the Change File locations and is therefore only relevant if the task's Apply Changes or Store Changes options are enabled.
The format of the reference file name is as follows:
<google_cloud_storage_target_endpoint_name><counter>.csv|json
Example:
MyGoogleCloudStorage00000001.csv
The counter suffix increases incrementally each time a new Reference File is generated (i.e. which occurs when the file reaches the maximum size defined in the General tab). Once a new Reference File has been generated, you can delete the old reference file(s) if required.
Whenever an Change Processing data file is created, a new row is added to the Reference File in the following format:
<Source_Table_Name>,<Data_Lake_Store_name>/<path>/<file_name>
Example:
employees,mydatalakestore/new/files/my.company/20170611-120144192.csv
Note that if the Run command after upload option in the Advanced tab is also enabled, the Reference File will be generated after the post-processing completes.